Skip to content

feat: [velox][iceberg] Add sequence number conflict resolution for positional deletes and deletion vectors (#16873)#16873

Open
apurva-meta wants to merge 5 commits intofacebookincubator:mainfrom
apurva-meta:export-D97530139
Open

feat: [velox][iceberg] Add sequence number conflict resolution for positional deletes and deletion vectors (#16873)#16873
apurva-meta wants to merge 5 commits intofacebookincubator:mainfrom
apurva-meta:export-D97530139

Conversation

@apurva-meta
Copy link

@apurva-meta apurva-meta commented Mar 21, 2026

Summary:

Extend the sequence number conflict resolution logic to positional deletes
and deletion vectors. Per the Iceberg spec:

  • Positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum
    (strictly less than, unlike equality deletes which use <=)
  • Sequence number 0 (V1 legacy) never skips

Changes:

  • IcebergSplitReader: Apply sequence number filtering for positional
    deletes and deletion vectors before passing to readers
  • IcebergReadFile: Store dataSequenceNumber for conflict resolution
  • Tests: Add sequence number conflict resolution tests for positional
    deletes and DVs

Differential Revision: D97530139

@netlify
Copy link

netlify bot commented Mar 21, 2026

Deploy Preview for meta-velox canceled.

Name Link
🔨 Latest commit c6b8d72
🔍 Latest deploy log https://app.netlify.com/projects/meta-velox/deploys/69be3b9685fe450008aee6a6

@meta-cla meta-cla bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Mar 21, 2026
@meta-codesync
Copy link

meta-codesync bot commented Mar 21, 2026

@apurva-meta has exported this pull request. If you are a Meta employee, you can view the originating Diff in D97530139.

apurva-meta added a commit to apurva-meta/velox that referenced this pull request Mar 21, 2026
…sitional deletes and deletion vectors (facebookincubator#16873)

Summary:

Extend the sequence number conflict resolution logic to positional deletes
and deletion vectors. Per the Iceberg spec:
- Positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum
  (strictly less than, unlike equality deletes which use <=)
- Sequence number 0 (V1 legacy) never skips

Changes:
- IcebergSplitReader: Apply sequence number filtering for positional
  deletes and deletion vectors before passing to readers
- IcebergReadFile: Store dataSequenceNumber for conflict resolution
- Tests: Add sequence number conflict resolution tests for positional
  deletes and DVs

Differential Revision: D97530139
@meta-codesync meta-codesync bot changed the title feat: [velox][iceberg] Add sequence number conflict resolution for positional deletes and deletion vectors feat: [velox][iceberg] Add sequence number conflict resolution for positional deletes and deletion vectors (#16873) Mar 21, 2026
@meta-codesync
Copy link

meta-codesync bot commented Mar 21, 2026

@apurva-meta has exported this pull request. If you are a Meta employee, you can view the originating Diff in D97530139.

…ix test file handle leaks (facebookincubator#16869)

Summary:

- Add explicit equality deletes NYI branch in prepareSplit()
- Improve VELOX_NYI error messages with descriptive content type info
- Fix FILE handle leaks in IcebergReadTest by extracting getTestFileSize() helper
- Minor doc comment formatting improvements in IcebergSplitReader.h

Differential Revision: D97530140
…okincubator#16870)

Summary:

Add deletion vector (DV) reader to the Velox Iceberg connector,
enabling Iceberg V3 spec support for row-level deletes.

Iceberg V3 replaces positional delete files with deletion vectors — compact
roaring bitmaps stored as blobs inside Puffin files. Compared to V2 positional
delete files, DVs are more compact and avoid sorted merge of multiple delete
files at read time.

Changes:
- IcebergDeleteFile.h: Add FileContent::kDeletionVector enum value
- DeletionVectorReader.h/cpp: New reader that loads a Puffin blob,
  deserializes the roaring bitmap portable format (array, bitset, and run
  containers), and sets bits in the deleteBitmap for the current batch range.
  No CRoaring dependency — self-contained parser.
- IcebergSplitReader.h: Add deletionVectorReaders_ member, include header
- IcebergSplitReader.cpp: Route kDeletionVector in prepareSplit(), apply
  DVs alongside positional deletes in next()
- BUCK: Add DeletionVectorReader library target

Differential Revision: D97530142
…ookincubator#16871)

Summary:

Implements Iceberg equality delete support for the Velox Iceberg connector.

Equality delete files contain rows with values for one or more columns
(identified by equalityFieldIds). A base data row is deleted if its values
match ALL specified columns of ANY row in the delete file.

The implementation:
- Adds EqualityDeleteFileReader that eagerly reads the entire delete file
  and builds an in-memory hash multimap of delete key tuples during
  construction.
- Wires EqualityDeleteFileReader into IcebergSplitReader::prepareSplit()
  to resolve equalityFieldIds to column names/types from the table schema,
  and into IcebergSplitReader::next() to apply post-read equality delete
  filtering with row compaction.
- Handles lazy vectors from file readers via loadedVector() before
  accessing values for hashing and comparison.
- Supports BOOLEAN, TINYINT, SMALLINT, INTEGER, BIGINT, REAL, DOUBLE,
  VARCHAR, VARBINARY, and TIMESTAMP column types.

Differential Revision: D97530141
…ality deletes (facebookincubator#16872)

Summary:

Implements Iceberg V2+ sequence number conflict resolution for equality
delete files. Per the Iceberg spec, an equality delete file should only
be applied to data files whose data sequence number is strictly less
than the delete file's data sequence number. This prevents concurrent
write conflicts where a delete file written concurrently with a data
file could incorrectly delete rows that were not intended to be deleted.

Changes:
- Added `dataSequenceNumber` field to `IcebergDeleteFile` struct with
  default value 0 (unassigned/legacy V1). When 0, sequence number
  filtering is disabled for backward compatibility.
- Added `dataSequenceNumber` field to `HiveIcebergSplit` to carry the
  base data file's sequence number.
- Updated `IcebergSplitReader::prepareSplit()` to skip equality delete
  files when `deleteFile.dataSequenceNumber <= split.dataSequenceNumber`
  (unless either is 0, which disables the check).
- Updated test constructor of `HiveIcebergSplit` to accept the new
  `dataSequenceNumber` parameter.

Differential Revision: D97530136
…sitional deletes and deletion vectors (facebookincubator#16873)

Summary:

Extend the sequence number conflict resolution logic to positional deletes
and deletion vectors. Per the Iceberg spec:
- Positional deletes and DVs skip when deleteFileSeqNum < dataFileSeqNum
  (strictly less than, unlike equality deletes which use <=)
- Sequence number 0 (V1 legacy) never skips

Changes:
- IcebergSplitReader: Apply sequence number filtering for positional
  deletes and deletion vectors before passing to readers
- IcebergReadFile: Store dataSequenceNumber for conflict resolution
- Tests: Add sequence number conflict resolution tests for positional
  deletes and DVs

Differential Revision: D97530139
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. fb-exported meta-exported

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant